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FOREWORD 



Evaluation of school progranis, is becoming more 'of a necessity for survival 
th/n a iMXury enjoyed only by af fluent , distri'Cts/ As financial resources 
dimiriisl>, decisions on how /to allocajte the available • funrfs must be made. 
While basic educational research prcivides much valuable inf ormation, that - * 
infopma;tion is'usually not^ the kind on which day-to-day decisions about " 
speqifl'c educational prog/^ams are based. Program evaluation, as perceived by 
the N^alifornia Evaluation Improvement' Project, is a means by: whicji useful* 
information is collected and analyzed by a* local educational agency for its, 
own use. ' ^ _ ; 



While .most educators, have had courses in testing and measurfemeni and ^Qja(& 
contact with educational research,, there has been little In their^l?lr'^^ng to. 
prepare them f or tconducting a systematic/eValuati-on of a local scliool or ' - 
classroom p^rogram. , Of course, evaluation has been going, on for 'iftany year^, 
but it has most frequently been at the Intuitive' level, with * little coTTsistency 
and little impact on the total educational program. '^v . 

California's response to, this problem has .been to ^develop a training -prbgram 
in basic evaluation concepts and skills, whicj^ i-s directed to the classroom* 
teachej, the , principal, the curriculum director, or program managgr whof ^ants ' 
to evaluate a local program to assist In ^.ocal 'cjeclsion /making . / * , 

One of the strengths of this training pi'ogram is ^that/it wa§ developed and' 
field tested' throughout California by a'-group f^f educators' whose backgrounds 
were primarily in the areas of* program pp.anning, curricijlum, \idministratibn, ' 
and Supervision. Evaluation specialists were us ed^ extensive!^ as consultants 
as the workshop training program was deve:|.oped, but the emphasis has been kept 
on how evaluation information gould help in ansWering questions raised by the . 
developers, whose orientation , was basically that* o£, program mada^efs. ' 

There is no magic formula to solve the' problems iifivolved/ In educating the / 
youth of America, but I hope that this training program in basic evaluation . \ 
concepts and skills will be useful to local schools and districts as they 
work tovAard improvement of the educatipnal process* - " \ 




WILSON RILES " . 

Superinftendent of Public * 
Instruction 



ERLC 




PREFACE 



Prpgfdm evaluation, through which a|[8chool or district evaluates its own 
prbg>i;^in tor its own purposes, is -did&erent frdm educational research. It is 
^iso different, friogi a state testing rprdgram and' firom gathering information 
required by .the state about achieVenfent levels in specially funded programs. 
^Program eyaluitioti at the scJ;ioc4 or district level should be something the 
school or distrJLct does^f Or , itself "ftor its own purposes rather than something 
art outsider' dp,(Evs or ±t^ , s ^ I ^ 

[ Program evaluation shoul4 be an .dntearal part of the program-planning process, 
j Prpvi'siohs sRjQuiLd be-^built intc/each-lprogram to^t information' that will 

i'iridicat'e progre'ss towards the* j)&og ram s objectives, the degree of implementa- 
j tion o'f the iyian, ' and bother itjformatipn required to make rational decisions 
I about the pjr^gram. ) | * * " . 

[Program evaluation is qf^littie yaluelunless setae use is made of its results'. 

jA part of the eval\iafton process ! ihkl|des identifying potential audiences for 
the evaluation report an4 finding; out What kind^ of information would be'' 
useful to 'thenj. Providing useful, trimely information to people who can use it 
Is one of the best' ways of ensuring |tn"^t the evaluation reports will be 
used. ■ 



These concepts are basic^ to the workgjyip materials that have been developed by 
th^ ^California Evaluation IioproveitfSxt" Project. The materials were designed to 
be as practical as possible for the edilcational practitioner, aud it is our 
hope that the reader will fiiid these] concepts useful and will be able tor apply 
them to future planning as well a^ t^ p|rograms that are currently in operation. 



* Alexander I. l^w^ - 
'Chief, Office ;of Prqgram 
Evaluation aad llese^irch V 




William H. Bronson , 

State Director, California 

Evaluation Improvement t'rojes'ct 



INTRODUCTION TO THE EVALUATION IMPROVEMENT PROGRAM 



Educational Testing Service (ETS) is pleased to have be^n selected by 
the California Department of Education ha publisher, under an exclusive 
license, of th^ California Evaluation Improvement Project (EIP). materials. 
These constitute a course of ins truction for individuals responsible fc^r 
school programs and for those who help educational administrators ascertain 
program effectiveness. At the time of initial publication^ spring ^1977, the. 
materials in EIP consist of the following: ; ' 

• ProRram^valuator^s Guide . The Guide is a basic manual which provides 
in consicNerable detail background knowledge on the steps involved in 
planning aKd carrying xiix^t a program evaluation. It is 'designed as a 
study guide 2Ms4.^^^1eap«Ung tool for use in inservi^ training. workshops 
for program evaluators. , . ' ^ 

^ • «. Workbook on Program Evaluation , The Workbook has two purposes^ It- 
can be put to use as a learning and instructional aid while dne 
masters the procedures, techniques, and methods of program evaluation. 
Used this way, it helpsthe practitioner summarize and put into* 
practice the subject-matter presented jLn the Program Evaluator^'s 
t' ^Hi^' best used, however, as a working notebook which the 

trained program evaluator ca,n use^for recording his or her plans as' 
they are made and for making notes on program and p^rogram evaluation 
activities andfcfVents during the course of th6 program yeatji Used 
in this way, it helps the program evaluator keep/ complete retiords of 
the important information related to "the program evaluation. It will 
probably be most useful when an inferim or end-of-year progr.am * ^ 
evaluation report has to be prepailred, for much tive" infoDrMtion 
needed at those "critical times ^i 11 already have been made Jfenatter 
of record 1^^ the Workbook. 

• Evaluation Trainer^s Guide . This volume is a companion to,^the 
Program^ Evaluator ''s Guides It supplies background and supporting 
materials for use by instructoC3 conducting program evaluation 
workshops. Graphic art is^pravided for visual aids iri support of., a 
variety of ;3ubjects. . ' 

See i^age xiv for information about obtaining these materials.' 

Publication '.yearly in 1*977 af the first of a continuing offering of EIP - 
mat'erials is oonsistent with ETS's long-term commitment\to help advance the 
art. of program evaluation in the elementary and secondary schools. The EIP \ 
materials are expected '^'to go through a number of printings under the ETS 
Imprimatur. Eath successive printing will be a revised edition. Here' we ask 
.the** help' and TiCDoperat ion -of ^ the readership. . ■' ^- 



As you, as a, program evaluation priictitioner', identify parts of any of these 
three wrfrks that could benefit from refinement and further development, or as 
ypu" think of experiences that would serve to illustrate points made in any of 
the stibject treatments, we hope that you will share your thoughts with , 

We would like to see the Evaluation Improvement Program subjected to its own 
prbgram evaluation by those who use its materials. We hope the evaluation 
will be formative, not sUmmative, in natjure, for it is our intention to 
cycle evaluative comment on each edition into significant improvements in 
later bnes. Present plans call for publication of the second edition, our 
first revision, late in 19^77, and constructively critical comments from 
practitioners calji be turned into refinements in print in very short order. 
Join with us make the EIP materials, initially well developed by the 
California Evaluation Improvement Project, even better as time goes on. The 
California Department of Education and Educational Testing Service have 
joined the common goal of making thp EI'P materials as '•practical and usffp&l 
as they can be made to be. . ' ^ 




>' I CONSULTANTS TO THE PROJEGT 
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GENERAL INSTRUCTIONS FOR THE EVALUATION TRAINER 



PThe Evaluation Improvement Program 



General 



TrairieV's Guide 
Instructions. - 1 



GENERAL INSTRUCTIONS ICR THE EVALUATION TRAINER 

Evaluation Improvement Program maf^rials in program evaluation have wo'rked 
'well over the past three years in workshops of up to 30 persWs and an 
instructional staff of two evaluation specialists. ^ ' 

An equally sat*i<sf ac tory arrangement has been to have an educator- 
generali^t to assume responsibility for instruction in evaluation purposes, 
evaluation- planning, data collection, and reporting and applying evaluation 
results as well as an evaluation specialist to assume responsibility for 
instruction in evaluation design, instruments, and data analysis. 

Where there are more than 30 participants it .tias been necessary to 
divide the part icipants ^into two groups and to provide a^ ins true tlpnal staff 
of four. With two groups, it is possible to, have dual tracks, one for those 
with substantial backgrounds in quantitative methods, the other for those 
with less knowledge of statistics, sampling, and design. 



; The importance of carrying out the learning exercise cannot be emphasized' 
too strongly. The exercises provide welcome opportunities for workshop 
' participants to interact with the subject matter and with each other.* So. 
long as the exercises move along at a good rate of speed, they add a great 
deal to the "take home" value of the workshop. Since post exercises 1 
are group activities, workshop space shoulife be arranged whenever poss ijfie > 
with thr^e to five persons at each table. Six groups of five persons feach 
has proven to be an effective way to promote good small group interac tion- and 
sharing of findings. * ^ - 

The nature of the subject matter provides many fine opportunities for 
lively discussions on sometimes controversial topics, discussions that can be 
sharpened by the instr\ictor through carefully framed questions or se^quences ofi^ 
questions.. Successful discussion periods have proven to be a matter of 
balancing time spent on such motiiva^ional- activity with time devoted to 
familiarization with and uses for the EIP snbjeqt matter." lother' discussions 
will result from questions by participants. It is important to give painstaking 
, attention to sucli questions, especially those that are^-oC- general interest. - 
It should also be emphasized, however, that discussions of .over a few minutes 
on points of clarification often cause a workshop to lag. Questions of a 
technical nature usually are best left for after the scheduled workshop 
session. ' ' 



Although the Kvaluatian Trainer's Guide Is keyed to the ProRram Kyaluator\s' 
Guide , the Workbook on Program Fivaluation h*as proven to be as useful in the 
/ teaching and learning process. Inspection -of the latter volume will show 

that its organization is compatible wlt.li those ot it\o other two volumes, and 
It Is an easy matter to find in ic matfriai to reinforce <uiy point that might 
arise. 
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Trainer's Gutde / 
General Ir^structions 



rime spent in 
foc/l^ter completion 
•qiu-ck page-flipping jj 
tion or concluding'/ 



»rting some of the Workbook activities during a yorkshop, 
home, has been found to have been time well spent. A' 
Lew of the Workbook makes an excellent initial orienta- 



The primary y^r poise- of the^ Evaluation Trainer^s Guide is to lend' support- 
to the trainer iw his or her presentations of workshop material^, with the 
Program Evaluator''fi Guide being the primary resource for participants. The 
Trainer ^s^ Guide is designed to be used at the lectern or head table. There 
are cross-references throughout to the appropriate pages of the Eval u^tor ^s 



Guide and to the Learuing Exercises, which should obviate any * searching fbt 
subject^. Material surrounded 'by boxes is suggest^ed text for Instructors' 
presentations. "Topics Covered " and ''Things to Stress" should be reviewed 
befi)re a presentation. "Suggestions for Teaching" can be followed in the 
course of a presentation. s ' ' 



Fpr the most part, the sequence of instructio 
one moves through the Evaluator's Guide . The text 
used with a set of 103 transparencies. Those used 
indexed on ,the back of each section's t^tle page, 
illusttations , charts, graphs, and other art work 
Guide , afid their page references are part of that 
transparencies JLocally. It is expected nonetheles 
'ehcies will be published as part of the Evaluation 
'will be available f oi;* , dist ribution - in summer 1977. 
equipment needed to conduct an EIP workshop includ 



n is' in ascending ord^r as 
has been written to be 
with each section are 
The, transparencies are of 
reproduced^ in the Evaluator 's 
index. They can be madsj into 
s that the prepared tratispar- 
Imprpvement Program and / 
Facilities, materials, and' 
e the following: , 



A conf ierence i:oom large enough for 30 persons clustered ^n groups of 
not more than 5 ' - . * 

Table^ space for participants so they can spread out thel,r papers and 
books ^ . 

One copy for each participant of the \ 
Program E valuator's Guide 
Workbook on Program Evaluation 

One copy for each instructor of the above publications and the 
Evaluation Trainer's Guide * ' . . 

Chalkboard or chart paper on easel , 
Chalk or wide-tip felt pens ' < 

Overhead projector and electrical connection . * 

Screen ' , - 

Blank a(::etat'es and transparency marking pens for participants to use 
in preparing their presentations 

Homemade trajispar^ncies of art work In Program Evaluator's Guid^ 6r 
prepared ttanspareuc ies from the publisher , ' 



^ Please give youv 'careful attention to the final paragraph on page viii; 
Send us- your comments for improvement of the EIP materials. Constructive 
criticism from practitioners will be reflected as refinements in the publica- 
tions in their next, edition* ' ■ 
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, * • Introduct^^on 

The purpos(l of ^ this sect iori^is ta-pr,esent* the varied -purposes and -require- A. 
ments qf evaluation as pe rceived'^by various^ audiences|^ (--^Iso gi\rps a 
brief overview of the evaluation|^rocess with a delin^eatiqn oi th(^Ynitial 
steps to be taken by the evaluatoT in the evaluation process. 

■In^roductibn to WdrPshop (a t page* A--1 of the Evaluator^s Guide ). 



TR.'^A-O .Projece transparency on screen^ while giying an introduct ion*\to 
the workshop. The trainer may develop the introduction as he wishes^^^'oX^, 
use t^e comments in-the box-es. ^ • ^ 

• • */ * ' • . 



The demand for evaluation of educational programs has^ greatly increased 
in recent years: The state legislatures have insisted upon evidence 
of achievem^jnt wt^en special funding has been given to^programs; the 
federa^ government makes similar demands for accountability wHen f^derdl 
funds are utilized by local srchool districts, and local .coimiuinities are 
becoming' more and more vociferoL|s in their desire to' be* apprised of 
demonstrated accomplishments of the local school programs. ^ 

As more and more districts adopt austere budgets, eacji program must be 
carefully scrutinized as tp its effectiveness. Some programs may be 
eliminated, otherb'" expanded , still others modified to take account of > ' 
their proven accomplishments and observed ^leficiencies . 

) ■ 

The Califo rnia Evaluation Improvement Project has been developed to 
assist program managers in making effective evaluation plans so that 
data may be {Droperly collected, analyzedjand reportied. The key goal 
of program^ evaluation is to provide good answers to the questions that 
legislators; the community, and ed'ucationa|. decision makers are asking 
about f)rograiiis carried out in the schoTj 
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^ Purposes of 



uation 



TR, A*l This transparency may be. used to initiate a discussion, of 
the various purposes and reasons ^or prt)gram evaluation. Possible 
questions^ for the trainer to ask after showing the transparency are as 
follows:^ ' • 

• What do you think the public reaction might have been to, this 
newspaper report? ' ■ * ^ js>' 

' • * Do you think the public had' anjP' other infotyatlon regarding pupil 
C Achi^emelit? ^ / ^ ^ 

' • ^If there had^been frequent and comp rehensiye^ rep or tiding to* the 
public, wptild^he report have caused /Such an uproar? 

■ ) 

Emphasize that one reason for evaluation is to keep the public informed. 

"/'■'/ 

Elicit othei>ftreasons pr purposes for evaluation from the audience. 
Have these resiJt6nse6 recorded on a ^ chalkboard or easel paper in front of 
the room bjy^/iO/ne of the participants. Qjroup the responses into four or 



n)0 re" maJor^^Cc 
communicating 



tegories of purposes rEor program evaluation, such as (1) - 
with the public; (2) providing Information to decision 
make.rs ; (3) irm^ roving existing programs ; and (4) providing satisfaction 
to the participant. la some situations, the . traine r may have to supplement 
the] respon^e^ f rism^th<2 audience. 



3. M9Ve into a discussion that emphasizes the groups served. by those 
purposes. Ask the {/artici^nts to suggest groups for- ^each 
areas follow^ 

Communicating with the Public 
Parents 

Business community 
Professional community 
Industrial community 
Ro tired c o mmu n i t y 
Special interest groups 



Examples 

.■ Mi 
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Providing Information for Decision Makers 

Classroom teachers r . , . « 

Priixhipal ' •' . V- 

District administrators 

Superintendent _ 

Board of education 

Improving Exist;inR Programs 

* . Classroom teaohers 
V* Principal 
' ^ ' District administrators 

Superlotendent ' » 

Board* of educatioa. . ; 
:t:*Ci t izens advisory coraiAittee 

; ' Providing Satisfaction t.d Participants 

^ Students 

Classroom teachers 

Instructional aides ^ 

Stress that each group has itsv own questions and therefore niay require 
different information from thq program evaluation. 

Ask the participants to take a tew minutes to road Purposes of Program 
Evaluation on pages A-1 through A-5 of the Guide , This will afford 
review and re info tc emeu t of .the preceding discussion. 

Overview of the Evaluation Process (at page A~5 of 'the Guide ) 
/TR, A-2 Project transparency while defining program evaluation. 



Progr'am Evaluation, a.s used in tlii.s workshop, is defined as the process 
of determining the value or ef f ec t Lvonesy of some activity for the •> 
purpose of decision makijiig. De,cision makers need LntormatU^in to doj-ermine 
whethor to continue, moclify, expand^or discontinue individual programs. 
Program Evaluation is seen as the source of tliat Lntormat Ion. 
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Avoid drawii*-but technical discussion and argument. Accept the fact 
that other definitions may be Valid. 

' .. • ^ 

Ask the participants to review the reading on definition starting on A-5. 

TR> A-3 State that the three major phases of evaluatipn are "PLi^-Conduct- 
Use fA-e of the' Guide ) . Suggest that typically more time is €pent on - , 

conducting than planning or using. Hopefully this workshop will demon- 

< ■ . v ' . 

strate tbe crucial rieed^for planning and using as well as conducting. 

.V ' • ' ' . - 

^ ■ * t •• " " - 

TR. A-4 Identify the eight steps of the evaluation process (A~6 of 
the' Guide ) 'and put . this "sect ion on Purposes and Requirements in perspec- 
tive' to the total prpgram evaluation arid . the program management- • 
^activitieswi.. , ' ' ' - y ; ' . ... ■ r*^-' 

5. TR. A~5 Briefly discuss the different types of evaluation data which may 
be collected and the di f f erent ways they may be utilized in the evaluation 
process. In presenting the transparency (A~7), cover the Product,, Process and, 
• Context portion so that only Formative and Summative are viewed. At this 
time distinguish between fbrmative and summative evaluation. Formative } 
evaluation is concerned witH the Implementation of the program and 
interim information on tfte students' progress ,towar:d attaininent * of & 
- objectives. pormative^evaluation tak6'^ , plac^ dur j|ng a program.' Siimmative 

evaluation looks at the final outcome — yhetlier or not s tated objiect ives 
were reached — and stakes place at the ^nd of .a program. 

Uncover the Product, Process and Content section. Differentiate 
between the three types of data tHat should be collected in an effective 
program evaliiation. x . ' 

• Product Refers to the program outcomes. Learner achievements of 

one kind or aiiothcr^are' the ejcpected^'outcome of every instructional 
^ -program. Theri^ could be exceptions such as in a program designed ■ 
to ^TTOduce materials, but the- emphasis in this workshop is on the - 
evaluation of instruction. » . 
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Process refers to the ac tivit tes used to suppori^ the achievement 
of expectfeid outcomes. This may include data such as instructional 
processes and inservice training ^ act ivities. . 

Cotitext refers to the environment in which the activities take ^ 
place. It may inclifde factors such as facilities, scheduling, 
rules and regulations , and sociological climates 



Ask the participants to review the discussion. on. Typ:es of Evaluation 
Data on pages A-7 and A~8. ' Call attention to the examples .given . 



Introduce Learning Exercise 1, Types of Evaluation Data, on pages A-25 
and 26 of the Guide . Ask each group of {{aj/t icipants.. to arrive dt 
consensus on the fouF items that illustrate product, process, or conte> 
Compare answers with those on the sheet on page A-26. Allow 'time for 
discussion. 



TR. A-6 Discuss briefly" the cycle of program management and evaiua^Rj.r 
the feedback froiti ope step to" another, the mutual dependGi*cy oi: .^11 of 
the steps, and the cen trail t^*' of evaluation in the procc-s (A-9) . ' 

Ask the participants to review the chart and discuss i: : on pages A-9 
and 10. Anj^wer que-stions and discuss further if required. 



Role of the Evalnator 



We have discussed the purposes^ of evaluation, two types of evaluation 

processes, and thq kinds of. evaluation data to be collected. The 

* j.» ' ■ " ' ■ ■ ■ 

next question to which we must address ourselves is **Who will' bo the 
person re^p^sible for the evaluation?"'' 



Questions to further stimulate a discussion on the role of the program 
evalnat or migh t be as ■ f o ilows : - 



• Should^ the program evaluator be from within the district or 
external to it? ^ . 

• "^What sources exist r both internally and externally - for assistance 

in /the conduct of Tprogfam evaluation? 

• What are the advantages of an internal program evaluator? iyisa3V;an-, 
Cages? . ^ _ ^ ...^ - / . ' * '. ■•■ 

• What, are the advantages of an external evaluator ? Disadvanta^s? 
(» What prof ei5.s4pnal and personal attributes should the prog.ram 

evaluator possess? ' ^ f 

' . ' ' - ■ ■ . ^ ' * K.;.^^.^ 

Call attention to the short reading on Role of the Program Evarliiator on 
pages A-11 and 12 of the Guide , , * 

' ' Initial Stei>s of th(^ Evqluator " • : > 



After the evaluator has been selected, he will need to address himself 
to certain questions before the development of evaluation standards. 



TR, A-7 Project, tlie transparency and briefly dipcuss each of the 
"Questions '(A-13) , ' ^ ' . . 

Suggest that one of the;next s teps of the program evaluator is to takt^ 
a close look at the needs statements, the goals and the objectives o£ 
the program. * 

Talk through 6r ask partLcipants to read pages A-13; 14, 15 in the, 
Guide (five minutes) . Revi^iw the main points again emphasizing the 
importance of the evaluator' s examining the needs as so^ssment, program 
goals and objectives, and judging how well they relate to the program 

activities. \' , / ' " 

J." > ■ ' . . 

Follow-iip ,q\ie.sL ions to the reading might include: 

• What should the evaluator look for when he or she-revLewM an 
t^xintlng program? . * ' 

\^ • Why is it important to know whether and how needs Were identified?: 

• Wliat might tlie^aluatcir seek to leaVu reviewing program" 
i,, glials and objectives prior to beginning with the evaluation 

p iann 1 ng ?, •« . ^ 
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• QuesiiX)hsv:^hout . needs^ techniques mpy ari«^'. The toplt isy^not * - 

.diseuss'd<l. at -length in this wo^rkshop ;^however , a good reference is An 
Analysis of. Needs Assessment Techniques for bducational FjLa'nninR at State . 
: Intermed iate^ and District Levels written by Belle Ru'ch Witkin, Ph.bi afid 
published ^Ijy the Alameda County School\ D.ef)artment in Hayward, California. 
Ileit is available through ERIC. (identdf;ieation; hum^ ED 108370)/ 

5. ^ TR> kr^% Distinguish betwaeVi goals and objectives. (See A-l5> ) 

,-6. TR, At9 Coinment briefly; on each of the criteria for working performance . \ 
objectives. (See.A-15 and.l6.) ^ ,y.^,;V* ' 

. J. TR, A-10 Ask tiie group to ideatify>;..jt^e six compon^ of the perf ormanc<^^^ 

; " objectives 'shown on the^^transpatency . 

■ ■ ■ ^ 

8. Introduce- Learni^t^ Exercise 2,. Identification of Measurable Objectives, ^ 

on pa'^ges A^27. and 28 of the' Guide . Ask table groups to arrive afc'W / 
consensus on tV^cli i teit\\.v Review and discuss with the total groiip when 
subgroups have finished. ' :^ . ■ 

PiP^loj^- .Ask pac^^ group to select one of the objectives in the 

i * . ^ : ■ S^^^^ does not sat;isfY the requirements of measurabili ty and"*^^ 

rewrite It according to the si:c criteria. 

9. Introduce Learning Exercise 3, JelYting Appropriate Object ivcs, /on pages 
A-29 - 38 of the Guide. Ask the pajt^ticlpahtti to arrange themselves in 
groups of five or six. Each group slioill d chocks e one of the four goals 

and match a number of appropriate ' objectives to itii Review All .wi te rials . ' - 
in the exercise before asking the parlLcipants to begiu. 



the Group Report: I^>Wn on page A-37 so tliat the group- ratings 



TR. 


A- 


'11 


TR. 


A- 


12 


TR. 


A- 


13 


TK. 


A- 


lA 



TR. 4-^15 
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A-16 


TR. 


A- 


-17 


TR. 


A- 


■18 


TR. 


A- 


■19 



Lng comparison tratraparency 



■ ^ 



10. Discuss the need for a review of program activities to insure their 
appropriateness for stated program objectives. (See Summary paragraph 
ab page A-16. ) , : 7; 

11. ' Introduce Learning Exercise 4, Matching Needs Statement to Program 
Objectives and Program Activities, at page A-39 of the Guide . Ask each 

small group to match ^the needs statements with appropriaite program 
objectives and activities by wzfitinfr the'number of tjie related needs 
.statement next to the matching program^ objective and program activity. 
- Review the results with ^hia total^group. ^ ♦ 

. ■ ^ • * ■' . ' '^Ji ■. , 

12. TR. A-20 Review topics covered to ,this poiat in order to maintairf the 
perspective of the total workshopl:^ > 

Requirements of Program; Evaluation (at page A-16 of the Gu.ide) ' 



Diff ererit_ decision makers— teachers, principals, superintendents- 
different evaluation-information requirements , and the questions # Hich 
must be considered in the early stages of evaluation planning. 



lac 



2» TR. A-21 Project the' transparency and discuss three ke^ questioris to 

be considered when determining the requirements of the particular program 
^valuation (A-16 -17). Ask if there are other questions that should be 

. _ J J ^ — J / 



considered . 



r 3. Talk through or ask the participants to review the reading on Requirements 
of Program Evaluation ^ on pages A-16 and 17 as a review of ' the foregoing 
discussion. . . 
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Review with the participants the suggested Evaluation Information Require- 
ments Form on pages A-I8'and 19 of the Guide . ^ 

Aslc the participants to review the short reading on End of Year Evaluation 
and Interim Evaluation on A-20.. Possible questions to ask after the 
reading has been completed are aa follows: 

• How do interim evaluation requirements differ from end-of - the>.year 
requirements? 

• Should the end-of- the-year evaluation include interim evaluation 
data? Why or why not? 

Next, review the part on Identifying Resources and Constra|ints, on A-20 
and 21. Ask: , .... — — ^ ^ 

I ' — ^ llZ: '^-^ ^ . 

• What evaluation resources are- likely to be available to a 
program? What are some likely constraints? 

Recall a constraint in a particular program that was satisfactorily 
resolved. 

Review with participants the summary of the tasks performed by a program 
evaluator at the purpose and requirements stage on A-22. 

' Call'dttention to the Checklist -^of the Steps in Determining Evaluation 
Purposes and Requirements on A-23 and 24. 

Present the Purpose section of the Workbook on Program Evaluation at 
pages 1-4, Emphasize its programmed instruction approach. 

' TR^__A;-22 Project this transparency a^ a bridge to the next section 
on Planning. . 
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Sectloa B 



PLANNING 



The Evaluation Improvement Program 



TRANSPARENCIES . ;^ ' ^ ' 

Transparencies Subjects Trainer's Guide Evaluator^s Guide 

TR. B-1 The' Evaluation Process B-1 B-1 

TR. B-2 First Four Planning Steps ' B-2^ ' ^ - B-1, 2, 3 

TR. B-3 . Five. Later Planning Steps ' B-2 B-2i4,5,6 

TR. B-'4 Review Evaluation Tlan B-5 . B-6 

TR. B-5 Planning Saves Time and Energy ^ B-6 B-9 
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LEARNING EXERCISE 
Planning for Assessment of Program Objectives B-11 



30 
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Introduction 



'ram 



The purpose' of tMs section is to present an outline' for a'gene^al progi 
evaluation plan. Emphasis is placed, on the need for careful planning pripr. 
to the implementation of the evaluation process and as the program to be 
evalfiated is being planned. • 



1. TR. B-1 Project transparency on screen while, giving an introduction to 
the section. a ^ 



In the last section, we discussed the different purposes and 
requirements of evaluation as per-celved by varidiis^^audiences, 
Thenjwe took a look/at the initial steps the program evaluator 
takes as he begins to develop a plan. Now we shall look at the 
sequence of steps essential to developing a total program 
evaluation plan. You will note that we are at Section B, 
Develop an Evaluation Plan. 



' 2. 



In the development of an evaluation plan, there are a number of 
steps that should be included. While there is no universal 
list or sequence that is best for, every situation, the steps 
we have developed should be of assistance as you develop a plan 
for program evaluation. Let's briefly review the steps. ' 



Suggestions for Teaching 

3 

The trainer should have become well acquainted with the discussion of steps, 
of evaluation planning on pages B-1 through 6 of the' Guide . During the 
presentation, this discussion should be referred to and each step expanded 
upon through classroom interaction. It is importAit, however, that the 
review of the steps be limited to no more than 10 minutes. If detailed / 
questions are asked, emphasize that this is« 
ich step will' come later. ^ 



overview and further discussion 
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— -J — 

Review the statement^ of^need 

Match the prograta goalff and objectives with the 
statements of need ri \ 

Determine the questions for which the evaluation 
design should provide relevant information 
Identify the general type of assessment instruments 
which will provide the most, appropriate kinds of - 
information . *, 

Determine in general terms the "most appropriate time 
for securing the needed information and who should 
do it 

Select the kinds of data analysis procedures which 
will be most meaningful for the^ data collected 
List the activities to be monitored 

4 

Identify the deadline 'dates for all required reports; 
identify all persons >jl a;^encies to receive reports 
Determine use to be madp >f each report or kind of 

r ■ 

data reported ' ' 



B-1,3 



B-A,5,6 



Program Evaluation Planning Form 



Turn to page B-2 of your Guide , There you will find the planning 
steps in chart form. This chart can be used by the program evaluator 
in developing a general evaluation plan. • 



Introduce Learning Exercise 5, Planning for Assessment of Program Objec- 
tives, on pages B-11 through 15 of the Guide . Ask' the participants to 
work in small groups, with each group devoting itself to one of the four 
objectives selected by Rosedale for its upcoming program evaluation. 

Ask each group to agree and record the title of "the program chosen, 
apparent' purposes, and a key audience in the appropriate spaces on page 
B-14. . - . ,9 , 
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4. , Briefly discuss the questions in the first two rows of the Program 

Evaluation Planning Form on B-15: ' 

Ptogram Objectives » * 

Evaluation Design . ' 

and then ask the, groups to apply the questions in these 'categories to 
their selected programs and record their responses in the apprropriate 
spaces on the work sheet on B-14, 

5. Next, discuss Y^^^ questions in the next two rows: 

Assessment Instruments ^ 
f Administration Dates and Personnel 

• ■ ■ 

After the brief discussion, ask the groups to complete the appropriate 
rows. Responses should lie the result of consensus among members of -the 
' ^ group. ' ! 

6. Briefly discuss the questions in the Data-Analysis' Techniques row and 
then ask the groups to complete that roi^^^^ .their work sheet. 

NOTE: Advise the participants that ^they should not be . 
concerned if they are unable to respond to all of the 
questions. By the end of the workshop, they will have 
^. sufficient information to do so. ^ 

"^1/ ^ . ■ • - . ■ ■ 

7. ^riefly discuss the questions in the next two rows:' . 

* Monitoring Program Activities 

. ■ . ■■■ .( 

Monitoring Dates and Personnel 

and ask the groups to complete these rows. 

8. Discuss the questions posed in the final rows: 

Key Reporting Dates 

Personnel or Agencies to Receive 
Reports . 

Use to be Made of Data or Reports 
and ask the groups to complete the work sheet. ^ 
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As tKe groups are working on the questions in each row of the work shee.t, ^ 
thej trainer should circulate among the groups and assi&t as necessary. 



frc 



9. ./As a culmination activity, ask a spokespersc^ from each of several groups 

to report in -turn on their results. The trainer should be alert to 

^ inappropriate plans that might be reported and indicate that more informa*- 
tion Xn this area will be given later, in the, workshop. 



Review of Preliminary Plans (at' B-6) 



1. 



1. 
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TR. B-4 We now have a preliminary program evaluation plan. The next 
step is to have the plan reviewed by all of those persons who ;are 
involved in the program or who may be interested in its evaluation. 
Staff members will be far more supportive of a program evaluation if, 

they feel they had some input durirfg its planning. Errors or m±sco?lcep- 

,' ■ ■ ■ - • y 

tions that may have gotten into the plan may be corrected. And, too, 

■ P ' , V ■ . 

the identification of potential problems in the planning stage may 

avoid difficulty during execution or opposition at the time of reporting 

results. , 



Evaluation Timeline (at B-7 and 8> 



Now let's take a look at planning the management aspects of a program 
evaluation. 

- > 

A timeline is developed for four purposes: 

• To use as a communication tool 

• To gain support and commitment among participants 

• To schedule designated activities 

• To monitor scheduled' activi*ties 

As a communication tool, the evaluation timeline becomes the calendar 
for all major activities. School personnel involved in or affected by 
evaluation activities need to be notified of forthcoming events as well 
as being included in the planning and scheduling of some of the evalua- 
tion activities. As a tool for helping to get the evaluation done, the 
timeline provides a concise picture for the participants and manager, of 
evaluation activities. \ 
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The timeline should show each major activity, the time-frame for 
accomplishing the activity, and the personnel involved in completion 
of the activity. The timeline also becomes a useful tool when 
evaluation events go' off schedule. Revision dates 'can be added and 
the effects of the revision on other events can be viewed. 

It is important to remember that timelines and activities generally 
go through revision throughout the year. Don't discard your timeline 
when> events or dates beg;Ln to change. Be flexible and chgrnce those 
sections of 'the timeline that have undergone revision, holding to the 
basic pattern. 



ReviUPth 



ReviS^the timeline form on page B-8 of the^ Guide . State that this is 
only one example and ask the participants to describe timeline charts 
with which they have worked. 

• With whom should timeline charts be shared during implementation 
of the program evaluation? 

Determining a*nd Obtaining Required Resources (at B-7 and 9) 



The next step. for the program evaluator is to ask: 

• What resources are required for implementing the program 
evaluation planned? ' - - 

• What resources are available? 

• How can descrepancies between resources required and resources 
available' be rectified? '.■ ' ^ 



Talk through of ask the participants to read Determining aod Obtaining 
Required Resources on pages B-7 and 9 of the Guide. 



\ 
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Checklist^ of 'Major St^ps " ; . * ' 

, ' ■ " 'A 

Call attention to the Checklist of the Major Steps Rlequired in Developing* 
a General Evaluation Plan on page B-10 of the Guide, * . 



Present the Needs Assessment, Goals ancl Objectives 3 and Data Req|piredents 
sections of the Workbook on Program Evaluation (pages 5-24) • If* time permits, 
go through the first steps of working with one of them. 



TR» B-5 This is the completion of thp section devoted to ^evelop- 
ing a general program evaluation plan. Planning your" evaluation 
carefully will not only save you time but also energy and will relieve 
you of any anxiety you may be fjeeling, , ' 

Our niext action will , be devoted to determining the appropriate evalua- 
tion design f a<^ddre^B'^ng the questions that' need to be answered 
about the program being evaluated. 
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TransDarencleB * 

TR. C-0 
TR, C^l 

TR, C-2 



TR,C-^3 



TR* 0-4 



TR, C-5 • 

TRf C-6 



TR,C-7 
TR. C-8 

.TR,G-9 

TR, C-10 
, TR, C-11 

TRf C-12- 
^ TR, C-13 
TR.C-14 
TR, C-15 



Subjects 
The Evaluation Process 

Are the Students Entering this 
Program Really functioning at 
Level Expected? 

At What Level Are Students 

Functionipg at/ the End of the 
Program? , ^.^ / 

How. Much Did Studeat^s Improye^i 
During 'the ;Prdgram? 

Bef dreV Intervention /Af ter 

;*How Much Did Students Improve 
During Program? 

How Db Nonprogram Students 

Identical with Program (1) 
, Students Compare with 
Regard to Growth? 
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How Do Nonprogram Students 
Identical with Program 
Students Compare with 
"ifegdrd^tx) Growth? 



(2) 



How Do Comparable Nonprogram- 
-Studfents Compare with Program! 
'Students? - 

How Do Program Student^ Compare 
with Same Norm Group? 

HpW Do Students' Achievements 
in .t1i.e ;New Program Compate 
with ..Their Past Peirf OFinarioe? 

Random Sample 

Tabie of Random Numbers 

Number of Stucfents 

Students for Qlassroom 
Observations ' 
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-3 
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c-6 V 

C-7 
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;c^2q:-7- 23 
,c- 26 

C-28 ' 
(S-31 ' 
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Ihtroductlon 



; .The purpose of this section is to present a useful ^ut nontechnical discus- / 
... sion of evaluation design. The background given is suflficient to'- the task of • 
:* selecting a design in a given set of circuras tances. The sect iorf ^Iso discusses 

sampling and emphay.zes sl^tuations in which the us> of sampling procediir^p . is 
^ appropriate. . « ' , ^ ' ■ , V ■ 



[ -. .. Topics Covered -in This Section 

• How to Select a Design 

-< • The JLevel at Which Students. Are Functioning 

• Gains Made During the Program 

• Gains Made irt Comparison to Expectations 

• Element^ of the Program Contributing b,Q Gains 

• ^Int'erpretability of Results \. 



• • Pitfalls to Avoid 



; • FaVtors to be Considered 

• Monitoring Ac^i;Uvitles 

• When to Use a Sataple 

• How-to Select a Random Sample 
. S^t^r.atif ied^ R|t;idom Sam^jles 

• « SaBiple Size 

• C^fie Studies 
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Things to ^jtress in This Section 

■ The b;asic question in program evaluation is whether the progtajoi resulted 
in improved leiati^ previous or current programs. (T-his 

po.i^nt should be periodically reinforced.) 



2. The program evaluajtor should work with th^e program manager while the progiram 

is in ■.'ttv^,:'*pianning stage.- .''^^^v v ' * f"-. '•'^ 



« 



■i 
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,3L. Design begins when you\knpw whaC^uestions will have trf^be answ^^ \ 
/ the end of the'evaluation. . •/ , ' . ; 

. . ' ' •■■•U.^* ■ . ■ 

4. The better designs involve some' kind of reference ♦ group or reference 
data. ; , < ■ 

. . ' ' '■ . ' «' ' "'^ ■ • 

5. Random ass 1^ of v Individual pupils to different programs .will db more 
• * to assure thatV fe^^ alike than u^in^ intact classes. 

6. Factors Which 'may influence the program evaluation results should' be 
ant icipated #in advance dnd taken into account in choia#^j^ (Evaluation * 
designs, ^ » 

7. The evaluation design must be chosen^ as soon as feasible — ideally 

when the program itself is planned ». hopefully before the program ' , ' 

starts y but certainly well befox^e data^^ has begun. ^' 



f"- ' ' ' 



r ^ i " y ^ 

;/ The question, of primary importance in program evaluation is:. How much 
more did pti^ils learn by participating in the , program than they woi^ld 
have lei^rhed without it? • ' 

To find out whether students have done better in a special, program than 
they would have done without it requires information about: * 
, 1.* How much students improved * , ■ 

2. An accurate estimate of how they would .Ka^^^ the 
■ program' .i.' ' ^ — '.^1 ' 
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, . " C-3 

I ' ■ ■ it 

( ■ 

Evaluation Designs 

t ' • \' ' 

SuRRestions for Teaching ,.f ' • 

1. JR. C-0 Tie evaluation design to' bthe|r parts of the evaluation process. 

2. Present the designs given on transparencies C-1 through C-11 and refer to 
the appropriate pages of the Guide as the presentation proceeds. 

Warm. Fueling — Stress the lack of objectivity in this type of 
, , ^ evaluation and that this does not constitute good program . , " 

evaluation (p. C-1). 
TR>C-2 } , . . 

TR, C-3} I^^icate that these do not constitute good program evaluation 
either (p. C-2) . 

TR C-4> Communicate the idea that these designs are better in terms of 
TR. C-5 } giving more in f onnat ion bu t they still do not answer the 

' questions about how worthwhile this program is (pages C-3 & 4) . 
TR, C-6 } ' ' 

TR.C-7 } : I 

JR/C-B } Of these alternative designs, stress that random assignment is 
"^R^ } to be preferred if at all possible (p. C-5 throup.h 8). 
TR.C-10 } 

m^c-ii} 



KLements oi the Program ContrlbuLLng to Cains 

<*) ' • , 

What to St res s (a t ( : 8 - 11 ) 

rndLcato that t actoi ial des L>;n^i sueli as theso give aclcled liUonnatLon it they 

^■nn he implemenred sur re.s.-. I n 1 ly . Tlu^ mi I L ot analysis can be either eutlie 

e la.ssi'Doms nv Individual students. Whatever units are seloeted, they musL be 

randomly asslgnod ti^ the various rolls In the table. It entire olassrooms 

art^ usinl as units ami there are two elassi'ooms por cell, lUou N==2 (rather 

< 

than the total unmbet ol students hi tlu, i^o la ss rot)ms ) . Students are 
countocl on_Ly_ U tlioy have been raiulonily assigned on an Individual basis to 
the various r Olid 1 1 Ions * 
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SugRestlons foj: /Teaching 

!• -Summarize the points given in the opening paragraphs on C-B. 

2, Ask participants to turn to page C-9 and look at the two factorial 
designs as you describe them. ) 

3, Have participants turn<;>to the design on page and explain the differ- 
ence in units of analysis (i.e., classrooms vs. students). 



Interpretability of Results 
3u^Rest ioivs for TeacM ng (at 0-12; 

This discussion and the one on Pitfalls to Avoid arc important to consider' 
if designs are to be properly carried out and misinterpretations avoided. 

I. Discuss each oi the points mentioned on 0-12. This material can be 

used to generate interesting diseussions since participants often wiii 
have had similar experiences that have tended to invalidate evaluation 
results. The point is that the program evaluator must be alert to 
. extraneous problems which can lead to erroneous conclusions about tlie 
effectiveness of a program. 

There are three sets of instructions under Pitfalls about wliich 
4uest:i.ons of toy arise. Tliose are discussed in 5,6, andli^'/, on C-14. if 
anyoiio (lui'stions thei;e instructions, here are tiio answers: • • 

(S) :i'he reason you should not use a selection test a's a pL*etest is 
.1 statlstieai artifact. Hecause of e v io£ inherent in ali 
nieasuiUMiUMit , extriMik^ groups wi I i always eontain siMue stutlents ' 
who haVi^ periormetl atypical ly well or atyplcaiiy pot^rly. U." 
you gave the )»,roup the same test tlie next day witli '-^^ i^'Veuiug 

i u.^it I uc i Ion, .some pupils wouLtl score much liigher (of ioWiU') 
than t ht^y did on the previous day. This Is ca i IihI "re^;ressli)u 
toward the mean." n dULerent tests are not used, results 
from pi'i'tesl-pi)sttest compariSi)iis will he contaminated. 
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(6) Therfe are two primary p^roblferas with grade equivalents: i. Grade 
equivalent "scores" do not qualify as score data; they do not meet 
the assumptions for t-tests, analysis of variance, etc. 2/Most' 
normative data are collected at one or two. specific times during 
the year. At other times during the school year, values are 
inferred by fitting a- curve to the actual data from those collections, 
This assumes that learning proceeds uniformly pver the school year. 
There is substantial research evidence that says this just is not 

so. - 

• •* 

(7) Computing gain scores, the difference between pro- and posttest 
on a single group^has s ta tisitical limitations due to differential 
growth rates, different test score reliabilities as a functio^n of ^ 
achievement 'level, and test floor effects. These factors work to ' 
invalidate results when inherent ial tes ts are used. There are two 
kinds of gain scores: raw gain scores and residual gain scores. A 
raw gain score is the difference .betwe.en pre- and posttest. A 
Ccsi4ual score is Lhe difference between a posttest score and some. 
estimated or expirtrted posttest score, where the expectation has 

been derived from the pretest scores using regression techniques. 
Raw gain scores systematically overcorrect the differences between 
program group and reference group; residual gain scores $iy y tema t Lcally 
undercorrect. 

Stress the kinds of L'aoinrs wliich must he considered in planning; a sound 
design. (See and 13,) This section can be used Lo generate a 

discussLon if the . t ralner wishes . Most participants can add to Lho 
List of fat:ti)rs if. gLven the opportunity. - ^ 



I'iually, stress the need id mtiuLLor program act LvL ties in order to assnrt^ 
imp LcMuetat Lou according; to plan. (See O I S . ) Not all K^^^^^ according ti>, 
plan uuLi^ss sonuu)iu» ha^i the specific job to sei* that: it y^n:^ done. 
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4. Proceed with Learning Exercise . 6, Evaluation Design, at pages C-16 
through 19'. There are good designs other * than the' solution presented. ^ , 

\, Fox example, a factorial design would be superior if it could be implemented. 

5. Summarize the main points on evaluation design before proceeding to sampling. 



Sampling 



ThjjlES_to_Stress (at C-20) 



1. Past experience indicates that some confusion exists about when to 
sample and when not to sample. This issue , is addressed in your opening 
introductory statement on the following page and in the discussion (How 
Large Should a Sample Be?) on page 'Ct9 of this .. Guide . Periodically 
reinforce the idea that sampling for program evaluation purposes should , 
be used primarily with large settings (i.e., parents , community members, 
large school districts) . Sampling within school programs is not usually 
feasible for pXpgram evaluation pu^trposes unless a factorial design is 
used or the number of students in the popu;[.at ion of interest is large. 
Withln-classroom sampling may be useful for certain purposes. 

2. . Sampling is a good way to get more information at less cost tibout a 

total program. 

'jr Using ci table of. random numbers is H simple procedure and the ~ - 

soundest one to use when drawing a sample. 

4. A stratified random sample gives greater assurance tliat the group 
^ stiid'ied wlLL be representative of the total popnlatlou. 
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When ihfonnation is needed on large numbers of persons, it is always" more 
economical to sample a portion of the total number on whom you would like 
. , to get the information. If sampling is properly done, you can infer what 
the- results would have been, had it been possible to obtain data on 
everyone. The decision tq sample for program evaluation must be ma<ip in 
relation to the size of the group on whom you want information. Sampling 
within classrooms or programs having relatively few .students is not 
defensible in serving program evaluation purposes Xh^ugh it may be useful • 
in providing Information that will help the teacher^make decisions about 
, day-to-day teaching " plans . If there ar^ large numbers of classr^ooms 
involved in a given program, then a sampling of students from each class- 
room may well be worth considering. The sampling of parents aiH community'" 
groups, however, is a practice almost all schools and districts will want 
- to consider. 



jiig gestlons for '^^ ^' ^^<^li_L n^ 



1. 



After the introduction above, the Introduction to Sampling at C-20-23 
in th'e Guide m^iy either be delivered in lecture form or talked through, 
or the participants may be asked to read it themselves. 

2. Int.rpduce Learning Kxerc Ltje '7 , Sampling" Cons Iderations, on pages C-24 and 
25 by reading the situation. Give participants the option of working 
alone or in groups. when most participants seem to. have finished, review 
Ltenu^ presented on comparison shoot and ask If anyone has additional 
po in ts to nuike. 

How t o Sel.oct- a RcUidoin Samp I o (at (:~26) 



1^* C-12 The method of selecting a random sam[iLo is not complicated, but 
tliero are some couuuou misconceptions as i.o how a random sample is drawn. 
The random selecL ion oi a sample must glvi* each person In tlie popnlation 
an e^jual chance to be drawn in the sample. 
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SufeRestjons ,for Teaching (at C-26) 



1. Following the introduction, demonstrate use of Table of Random Numbers as 
on C-26 and 27. ' . . . 
TR. C-13 Table of Random Numbers, Use to demonstrate hpw numbers are 
selected. ' . ' . . 

2. Introduce Learning Exercise 8, Random Sampling, at pages C-28 
TR. C-14 through 31, and give participants 5 to 10 minutes to complete 

it. They may work together if they wish to*do so. As they 
TR-> C-15 complete the task, have them compare their selections to those 
given on the comparison sheet. Help clarify misunderstandings. 



Another way to handle the problem iS^ to draw a stratified random sample^ 
In a stratified random sample, the population is first divided into 
categories or strata and then random samples are selected for each 
category or stratum. The more catagories that you include, the less you 
have to depend on randomization to handle the extraneous or uncontrolled 
factors,. This is so since the units withii) a randomly sampled stratum will 
be alike on' the category selected for stratifying. 



SuRRCsLions for Teaching (at C-32) 

1. Example: Ask participants to turn to page-C-'3^ in th^ir Guide , 
Work'' tlirougU the successive divisions and subdivisions to the final 
str<it if icat ion by sex, number of periods, and grade level on Cl-33. 

2. Talk tlirough the reprise at tlie end of the example. As an' optional 
d Lsciission,* yon might ask how the. group would stratify a community 

if they were to do a survey. Poss ible strat if icat Lou variables might 
be: . ^ / 

a. occupational level of parent or otlier partLcipaut, 
h. , ch L 1 (1 run Ln 'sc Uoo I v^v. chl Ulreii not in school 
c. hlllu^;ual vs. mono I Lngiia I 



EKLC 



Trainer's Guide 
C-9 



3. Example: Similarly, on C-33 and 34 „ work through the stratified- 
random selection of test items. Be sure to note that the number of 
items in the cell do not have to be equal. 

4. End this discussdon with a brief mention of cluster sampling as at 
the bottom of C-34, • • ' 

■ . ' ' ^ ,/ J . " ' 

How Large Should a Sample Be? (at C-35) 



Sampling students or student groupings within large schools or districts 
for program evaluation purposes is practical; but for smaller schools and . ' 
districts it is not. However, sampling of parent or community-groups is- 
practical for all except the smallest of school settings. The crucial 
factors are the size of the population and the ^ount of error that 
population size imposes. The amount of error the evaluator is willing to 
tolerate- is what determines the practicality of using a sample. "Population^* 
in this sensb refers to the group for whom you want information — it may be 
all fifth graders, or aJH. parents of secondary students enrolled in 
uoncoll^^ge preparatory curricula, or all adults of voting age in the 
community. . . 



Suggestions for Teaching 

1. Present the example in the text at pages C-35 and the Table for Determining 
Sample Size at C~36. 

Two Case Studies on Sampling (at C-'M^ 

SugRCstions for TeachiiiR 

1. CcMitor a discussion on one oC the two dxiw. Stuclios at pa^^es C-37 through 40. 

2. It: t LmC" short , refer to the other Ln passing tor later review. 
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Section D 



INSTRUMENTS 



The Evaluation Improvement Prograiil 



imi^ 



TRANSPARENCIES 



Transparencies ^ 



Subjects 
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E V a 1 u a t o r ' sJGuide 



TR. D-1 The Evaluation Process D-4 

TR,D-2 ; General Characteristics ^ 

of Standardized Instruments D-4 

Tr;,D-3 Reliable But Not Valid D-4 

TR. D-4 Neither Reliable nor Valid D-4' 

TR.D-5 Reliable and Valid ^ D-4; 

.TR, D-6 What is Being Measured? /D-4 

TR.D-7 What Kinds of Instruments 

Will Be Used? D-5 

TR*D-8 What Kinds of Item Types. Are There? D-5 

TR,D-9 . \ What Kinds of Scores Are There? D-5 

■'TR,D-10 Raw Score, Grade Equivalent' ^ D-5 

TR#D-11 Major Uses of the Interview Technique D-7 

'•rRrD-12 ^ Likert-Type Item f rpm a Teacher 

■ Questionnaire D-7 

TR.D-13 Summary of Questionnaire 

Item Structure , D-7 

TR,D-14 Observation Record D-8 

TR,D-15 Assuring Rater Reliability on 

\ ,. Observation Instruments D-8 

TR,D-16 -Objective 1 Bilingual-Bicultural ^ 

Unders tanding ' D-10 ' 

TR. D-17 Objective 2 Reading Comprehension D-10 

.TR.D-18 Objective '3 Language Usage D-10 

.TR.D-i9 Objective 4 Remedial , Mathematics D-10 

TR^D-20 Instrument Development ' D-il 

TR..D-2i 'Summary of Basic Evaluation 

Instruments D-11 



D-13 

D-13 
D-14 
D-15 
/f)-15 and 16 
D-20.. 

D-20 
D-21 • 
D-21 
D-23 
D-28 



p-36 ' 

D-42 
D-53 

D-57 

\ 

D-69 

D-6^) 

D-69 
D-69 . 
'D-78 

D-80 



/ 



LEARNING KXHRCLSLS 

9 Reliability and Validity • D-17 - 19 

10 / * Judj-; Lng : Items ' , 

11 Criticizing a Classroom Observation Instrument D-39 - 62 

12 Selecting Norm-Relerenced Tests ' D-6^j - 73 
1.3 Selecting Appropr Lat;t> Instruments ^ . D-74 - 75 
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Introduction 



f 

/ 



The central purpose of this section is to help make workshop participants 
aware of where to turn for help in idehtifying ^and s^.lectitig instruments to 
use and when to develop needed ins truments locally . Beyond that, there is an 
orientation to educational measurement and a survey, of each of several types' 
of instruments, " ' 

Topics Covered in This Section 
•\ Scenarios , to Set the Stage 

• Standardization / 

• Reliability and Validity. . 

• Norm-referenced Tests 

• Criterion-referenced Tests ^ 

• Questionnaires \ y 

• Observational Techniques and Instruments ■ . ' ^ 

• Other Behaviors ' 

• Where to. Find Existing Instruments 

• The Development of Instruments 

Things to Stress in This Section , . . . . ; . , - - 

1. Program evaluation needs to be planned in-order to prevent ' the kinds of 
things tliat happen in the scenarios. 

2. The careful selection of .projgram and program evaluation object Lves and of 
the evaluation design must precede determinations rcgai-ding assessment 
instruments. 
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Suggestions for Teaching 



SCENARIOS, 



-V- 



Nbw that you have .seen the* kinds of qu^^st ions that must be addressed -in 
the planning stages", wfe ar^ goin^ to. take time, through a set df three 
scenarios, to show what often happens in school districts. Copies of the 
scenarios are in your manuals, pages D-1 through D-8. They illustrate 
sohie important ;g(?n^ral principles of evaluation which we will discuss 
af terward, " , . * 

The first scenario is entitled The Repof:t : Useful to Whom and for 
What? Suppose that you are the program evaluator of a reading program; 
waiting to see the principal (identify principal) of thfe spl^Qol who is • 
,^.,busy • talkving to Mrs. Smith (indic<ite Mrs. Smith). You know Mrs. Smith. 
She is a reading teacher and you have been in her classroom once or 
twice. Your impression is that she is a very outspoken type of person and 
that she often talks a great deal. * , 



The door of the principal's office is no t closed^ and , you hear the following 
conversation. ^■' ^ > 



/ 



1. At least a half-day before th is point is reached in the workshop, announce 
the three skits, describe the characters aad select persons to play each 

* part. The assignment of persons to specific roles is crucial to the 

success of the scenarios. Previous experience has shown the skits to be., 
highly successful or very dull, depending upon the spirit of the cast. 
Not everypne enjoys role playing. Select the most out-going persons for , 
the major pcirts (Mrs. Smith, the Principal, Chairman, Mr. Worth, Mr. 
. - Fairchild) . If ycni hcwe worked with the group previously , ,you will know 
who will do the best Job. If not, check with the trainer who has worked 
with the group on previous sections in order to line up good protipects 
tor the parts. ^ 
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Discussion can follow each scenario or be held after all three have been 
done. At any rate, this material is generally a good "warm iipj* and it is 
fairly easy to encourage group ,distussion about the. various messages j 
contained in the scenarios. A list of points illustrated in the .scenarios 
follows the last one, at page D~9.in the Guide . It should generally be 
used as a check to see if the participants have spontaneously recognized 

the significant issues. - 

• . ■ • ■' •■■ , ''- ■ * 

Overview and Basic Considerations (at D-li of the Guid^ ) 



TfiinRS to Stress in This Section ' \. 

!• Periodically mention the desirability, of using, iriultiple measures for 
a given*program objective. " / ^ , 

2. The program evaluator should spend^ime searching in the JLibrary , 
familiarizing himself witH local resources. 

3. Note the broad definitron of ^^).rfi^^s>^ is. There are 
standardized questionnaires/and observation jrecords as well as 

'achievement tests. 



The remainder of the workshpp is desig^ned to help -you avoid the pitfalls 
illustrated in the scenarios. We have already talked about determini-ng 
purpoaea and requirements for program evaluation as well as the development 
of an overall plan , and the program evaluation design you might plan to 
use. This section will -concentrate on the instruments used to obtain data on 
program effectiveness. 
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SugResj:i,ons for, Teaching 

1, Begin with introductory t ransparancie^ 



TR 


D-1 


TR 


1 

D-2 


TR 


D-3} 


TR 


D-4} 


TR 


D-5V 



For definitions and discussion^ of reliability and validity (pages D-13 
through 16) . . 



2. Direct participants to Guidelines on Reliability, and Validity^ Coefficients 
on page D-16. ^ 

3, Introduce Learning Exercise 9, Reliability apd Validity, at D-17 thtough 19 
of the Guide « . , * 

^ ...... .-v. ^ : .. .... ... . , 

Some participants have trouble trying to handle the nondef initive answers 
given to this oxorciso. Ac tually the point of this exercise is\that not al 1 
is cut and dried or black and white in instruniont selection. You cannot 
always blindly follow a sot of rules. There is a point Where you must use 
your own Judgment and common senses. In ^ome cases, more information is needed^ 
before a' rational Judgmoiit can be made. It is important Cor the , program 
evalnat'or to recognize this and to try to find the addi tignal information 
needed. , _ ^ 



Types oL Assessment Instruments (at iWZO Oi the (Kildjj ) 

Achievement Teats 



TR> D-6 The siMection oL assessnuMit Instruments beg iuij. with thi> general 
(jnestioup. oL what, is to be measured. A soi^^what arbitrary, hut tuiuvenlent, 
clashs if Li'at lihi is p^e^:oilt:ed here. The first step is l:o think aht)ut what 
U is you want t d measure. ' 



I 



ae h iovtniieut 
at t I tudes > 

int e r.ic* t ion am ou>» persons 
othi' I behdV i nvi\ • * 
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SuRRestions for Teaching , : " ^ 

.1. Then follow in sequence, with: , I; - 
TR.D^7 Kinds of ins t rumeht s ; : . 
.-, TR/D-8 , Kinds of iter^H / \ ■ - 

■ TR>D-9 Kinds of 'scores ■ » . . V ^. . 

TR. D-10 Briefly review the table ^t D-25 on Types of Test Scores. 

3. Compare norm-referenced 'tests with*criteriou-ref eren(!ed test'd. <D-'22^arid 23). ' 
^'Cr iterion-ref erence,d h?as become a catchall tqmi, i;ibt only for measures 
appropriately describ^id by it/; but' also for measures perhaps more " 
precisely defined as content-(or \objectives~ or domain-) referenced or as 
cons true t --referenced. 

It is probably not necessary 'and may be undesirable to try to make 
these distinctions ^irh a workshop groupVv 'However, should participants 
inquire, you *mi^gh t . use. these examples : 

'a. *'Cr iterion-reCerehced*' ineans there is a defluiti^ pass-fail point 4-; 
such as in thg not.ring .of mastery scores » 

b. With content- (or objectives- or domain-) referenced measure3, no > 
\such external pass-fail requirements are set', but agreement has 
been reached on ob ject Ives , in a , g L ven domain, and the content has 
been adequately defined. ScDies on the InstrutiK^it have meaning in 
. ■ their own right witliont reference to external criteria or.noriikj. 

The National Assessment of Kducat i ona I I'rogress r^pifti;;8 this type 
of measurement. 

c- Cons t rue t~ refe reuced measurement Ls used wlien . [Vl t -^8 important 
to i:haract i»r i ze luilivi duals in m^rms of fumia menial trails 
, (e.g., i nte I I igeure, neni;i)t ( c Ituu) or to iharactiu* ize s ituatioi»s 

in term;; .of i uvulcuu^n^a I propert^ tv-; '(e.g., I i he ra I , to tal i la r (an) . 




AiKlofdou, -S'. n< ."yia 1 I , S. , Murphy, R.r, . l'.iu'yc4ll|MH.t<-i ot i-iliuMl 1 oua 
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Emphasize the point that aj norm-referenced grade equivalent score is a 
measure of central tendency. 

Summarize the common types of scores found on norm-referenced tests 
(D-21 and 25). Spend sufficient time on discussion of the table on D-25. 
An understanding of properties of different kinds of scores is essential 
to the selection of appropriate descriptive statistics and- inferential 
tests. Typically, the raw score is the best one to consider when using 
inferential tests for score data. Neither grade equivalents nor percentiles 
are score data. They are ordered data. 

Go on to criterion-referenced teste and draw the distinction between 
class mastery (D-23) and student mastery CD-24) . ^ 

Review some souWes where criterion-referenced tests may be found (e.g., 
table on D-26). ' ' ' ■ ' 



- Questionnaires (starting at D-27) 
Things to Stress in This Section 

1. Look for existing ins truments ^before developing new ones. 

2. Decisions about how the questionnaire is used to collect data must be 
carefully made'. Higher response rates can be achieved if instru- ' 
ments are administered in a captive group. But this may not give an 
unbiased sample ofr responses. Mailed questionnaires are easy to distribute 
but difficult to get returned. Special populations or unusual problems 
may necessitate one-to-onie interviews. 



When attitudes are to be measured, frequently some kind of questionnaire 
is used. There is a wide variety of questionnaire item types and 
techniques .. from which the program evaluator may choose. Knowing the range 
of possibilities and the advantages and disadvantages of each makes the 
task easier for the evaluator. ' . . 
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SuKRestions for Teaching , 

!• Introduce the questionnaire as a widely used means of collectitig various 
types of survey information. .Distinguish between surveys used in a 
group-response situation or sent home with students or mailed and those 
used aa^^ guide to conduct an interview. 

2. TR. D-11 Summarize where interviews are most useful. 

3. Have participants read the section on Guidelines on Review and Selectiqn 
on pages D-28 and 29. 

A. Discuss open-ended items and how they are categorized (D-30 through 34).. 

5. Have jpart icipants turn to page D-34 and summarize the advantage^ 'and 
disadvantages of open-ended items in questionnaires. 

6. Have part icipants follow in their Guide as you present the various kinds 
of object^ive items for use in questionna^^res (D-34 through 41). 

7. Go into some detail on ratirig scales (D-36 and 37). 

TR. D-12 This is an example of Likert Scales. Stress the need to^ 
define scale points. 

8. Introduce the Kropp-Verner Scale (D-37 and 38). This is an example of a ^ 
Thurstone Scale in which a seriels of value statements have been empirically 
assigned numerical weights that correspond to their judged worth or . 
value. On page D-39, which illustrates the scoring the Kropp-Verner 
Scale, tallies in Column 2 are shown as frequencies in Column 3. Column 

4 gives the numerical vdlue assigned to each statement on the basis of 
previously judged value in the process of standardization. This value 
can be used to obtain individual scores on persons by adding the value for 
each statement checked and then computing the median. Howeyer, group 
interpretation can be made simply by computing the median of the total 
group frequencies. 

9. Discuss how rankings differ from ratings. Refer to these two variations 
of the same item (D-40 and 41). Review with the group or have them do so 
individually. 

10. TR« D-13 Summarize the item structures for questionnaires (D-42). 

11. Introduce Learning Exercise 10, Judging Items; at D-43, and let partici- 
pants work in small groups. After about 10 minutes, review the results. 
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observational Techniques and Instruments (at D-48) 
Things to Stress in This Section 

1. If observational techniques are to be used, make sure they are sufficiently 
structured and that persons who use them have been sufficiently trained. 
Never use an observational method in program evaluation unless you have 
first checked the interrater tefliabil ity . 



While both achievement tests and quotes t ionnaires can give valuable informa- 
tion for program evaluation, there are certain kinds of needed information 
that, cannot be obtained from them. Observational techniques and instruments 
^or 'recording observations provide an added dimension. As with any other 
type of assessment instrument, there are both advantages and disadvantages 
in us ing them. ' . 



( 



Suftgestions for Teaching 

1. Ask participants to turn to page D-48 and follow table giving advantages 
and' disac^vant ages of observational techniques. 

2i Introduce Example 1 on how to observe quest ion-answe r-feedback sequences 
in classrooms (D-49, thro^i^=K^51) . 

3. TR^D-14 Introduce Example 2 on -measuring interaction among students (D-51 
through 54) . 

4. Ref er Cbr ief ly to D-54 through ^6 on the amount of detail required for 
reliable use of a sys.tem such as this. 

5. TR^D-15 Explain how to. assure interrater reliability (D-57). 

6. Summarize steps " in selection of an- observation instrument (D"^57 and 58). 

7. Introduce Learning Exei?cise 11, Criticizing a Classroom Observation 
Instrument, at D-59 through 62. Have participants work in small groups. 
Discuss the results after 5 to 10 minutes. 
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Other Typeri of Instruments (D-63) 



The last type of instrument to be discussed is the unobtrusive measure. 
The greatest advantage in using this type ot measure is that data collec- 
tion procedures do not influence the results. Data are collected without 
knowledge of the Individuals involved and sometimes* can be drawn from 
existing information files. Such things as attendance records; notations 
^ of awards, ;ci tat ions, and honors; conventional library records ; his tories 
of dropouts and transiency provide raw data for unobtrusive measures, 

~ ~ ^ ~ • T ^ ~~ ~ r 

Suf^Restlons for Teaching ' ^ • I - 

1. Read a few indi ca tors from the listings at D-64, D-65, and D-66 to 
illustrate a range of unobtrusive measures. 

Sources of Information about Instruments 
and the Selection Process (D-66) 



^"^os' Seventh Mental Measurements Yearbook is the best known source of 
published assessment instruments, but it is now somewhat dated. The 
references listed here give fairly comprehensive coverage over a wide 
range of instrument types. 



Suggestions for Teaching ' ' . 

Have participants turn to page D-67 and look at the table that shows 
where to locate information about tests- Discuss each briefly and show 
examples if you have thera. 

2. Ask participants to turn to page D-68 and follow Factors to" Consider in 
Selectin>; Instruments while you briefly summarize them. 
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3. Introduce Learning Exercise 12, Selecting Norm--Ref erenced Tests (See D- 69 
through 73)* After ^ short period of work, go over selections. 



TR. D-16. 



TR> D-17 



TiUD-18 



TR> D-19 . 




Tnofruments 

None appropriate. Instruments to be 
developed. 

Cooperative Test of Basic Skilig 
Cooperative Primary Tests 
Gates-MacGinitie Reading Tests 
Metropolitan Achievement Tests 

Circus 

Test of Basic Experiences 

Stanford Early School Achievemerit Test 

Conrprehensive -Tests of feasic. Skills 
Metropolitan Achievement Tests 



Using Existing tnr ^-Tn^nts or Using Your Own (D-76) 



Every type of instrument that hA been discussed in this section is 
accessible somewhere. Xhe progrln evaluator who spends time searching for 
instruments that will meet his program needs win be far ahead of the one 
who decides to launch a school-wide or district-^wide effort to develop 
locally made Q^^^^^^^n-ref erenced tests, questionnaires, or observation 
records. . . 



# 
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S uRAeations for Teaching 

Ask participants to read p.ages D-76 through D-79/ 

2. TR, D-20 When participants have finished, discuss . steps in instrument 
development (D-78) , ^ . 

3. Coiranent on need to establish reliability and validity of locally 
developed ins truments. 




Section Review 

Su ggestions for Teaching 

!• Sum}pari?:e the seven review points on D-79, 

2. Introduce Learning Exercise 13, Selecting Appropriate Instruments (see 

D-74 and 75). Ask participants to work in^ small groups. Discuss exercise 
after a short period. 



Present Instruments sec^n of Workbook on PrQ p;r am Evaluation , pages 
35-38. Emphasize importance of matching instrument to objectives to, be 



Ad- 



measured. 

TR, D-2J. Sujnmary of evaluation instruments (D-80) 
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TRANSEARENCIES 



Transparencies Su ^ects . Tralner^s Guide Evalu a tor^s Guide 



TR. E-O ^ The Evaluation Process . ' E-2. E-2 

■ ^ ■ • • ■" s 
TR»E-1. . Make Arrangements with 

School/Program Personnel E-2 E-2 



■ * LEARNING EXERCISES 
14 Planning for Data Collection ^ E-12 - E-14 



r . • . ' 
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' . * f ♦ • 

•.■ • ■ r* ' •■ ^ ' : 

Introduction 

' "■ ' ' ' ■' ' ' , 

The central purpose of this sect iofT^is to • stress the importance of the 
data-collection function in. the program ^evaluation process and of careful 
planning and advance preparation to be certain tKat all the people involved 
are ready at appropriate times to carry out their tasks. Things the program 
^valuator needs to do are covered, as. are the monitoring proceciures needed to 
give assurance that datd are collected as planned. ;iv 

Things to Stress in This Section 

1. Those points shown in TR. E-l 

2. Data collection requires'' careful management. The best design can be 
totally destroyed if data are improperly collected. 

3. Cooperation in data collection depends on perceived benefits to students, 
staff, and administration. The more apparent it is to all that the . 
program evaluation will meet their needs, the more confidence one may 
have that the data collected will be viable. . 

4. Decide who y^ur data collectors will be and plan a careful inservice 
training program for them. Even veteran test administrators can benefit 
from a periodic refresher course. If an instrument is being used for the 
first time in a school, go over the instrument and directions for its 
administration in detail with the data collectors. 'Never rely on written 
ins tructions alone. ^ 

5. Set schedules which be free of interruptions. ' 

/■ 

6. Monitor the collection of data. Check to see that everything is going 
as planned. 
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TR. E~0 



in the last section, which discussed the selection of instruments, the 
reliability and validity of the instruments was stressed. No matter how 
reliable and valid an instrument is, its usefulness can bie completely 
destroyed by carelessness in the collection and handling of data. The 
need for detailed data collection procedures must be emphasized. 

The eight s^ST^s of evaluation are interdependent: An evaluation is only 
as strong as its weakest element. Consider a few of the more obvious 
faulty data collection problems. 



SuRRestions for Teaching ' 

1. TR. E-0 Tie data collection to other parts of the evaluation process. 

2. TR. E-1 Introduce this section and then have participants scan E-1 
through E-6. ^ . 

3. Use the anecdote on E-6 to initiate a group discusbion of similar exper- 
iences . ' • . 

4. Distinguish between methods used to collect data from groups of persons 
and to collect survey data from individuals.. 

5. Ask participants to turn to page E-8 while you summarize the checklist 
for evaluators. ^ 

6. Review the sequence of data collectors' ac1:ivities presented on E-9. 

; 7. Collection of survey data (E-lO and 11) . Stress the need ioX achieving 
high percent returns and go over methods of obtaining them. 

8. Introduce Learning Exercise 14, Planning for Data Collection C!E~12 
through 14). Have participants work in small groups; Summarize after a 
short period of work at tables. 

9. Give careful attention to monitoring program processes and activities 
(see E-15 and 16). Point out two versions of forms for recording and 
monitoring information (E-17 and 18). » 
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F-ti ' 
F-2 
F-3 

F-4 
F-5 
F-6 

F-7 

F-8 

F-9' 

F-10 

F-11 

F-IZ 
F-13 
F-14 

F^15 
F-16 

F-i7 
F-18 



TR, F-19 



TR, 
TR, 

TR, 

TR, 

TR, 



F-20 
F-21 

F-22 

F-2 3 

F-^24 



The Evaluation Process 

Types of Statistics /. . 

Combining Results 

Frequency and Central 
Tendency 

Number of Perfl|j^ns 

Blmodal Distribution 

Results of Alzhievemetit Test 
(N«25) 

Mean, Median, and Mode ii^ 
Skewed Distributions 

Cases I, II, III: . Same Mean 
Varying Range . 

Case I: Standard Deviation/ ^ 
Score Spread 

Case II: Standard Deviation/ 
Score Spread 

Case III: Standard Deviation/ 
Score Spread 

Normal Distribution Curve 

\ 

Distributions Not Overlapping . 

Distribution Slightly 
Overlapping 

Overlapping Distributions 

Large and Small Standard^ 
Deviations 

Three Variables, Eight Groups 

Ranking from Scores j(Two 
Equal Scores 

Ranking from Scores (Three 
Equal Scores) 

Tests for Ordered Data 

Observed and Expected 
Responses 

Significance Levels for 
Chi Square • 

Responses on Tax Increase 
- Issue ^ 



Calculation of Chi Square 



.F-2 
F-2, 
F-3 

F^4 
F-5 
F-5 

F-5 

F-5 

F-5 

F-5 

F-5 

F-5 
F-5 
F-6 

F-6 
F-6 

F-6 . 
F-9 

F-10 

F-10 
F-11 

F-12 

F-12 

F-12 
F-1,2 



F-1 
F-1 

F-2 and 3 

F-4 
F-5 
F-6 

F-7 

F-9 

F-9 

F-10 And 11 

F-10 and 11 

F-10 and 11 
F-12 

F-14 
F-15 

F-15 

F-20 and 21 

F-22 

F-22 
■ ,F-22 - 24 

F-25 

. F-26" 

F-2 7 ...> 
F-25 ,., . 
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Trainee's Guldi 



This section is designed to help workshop participants feel more comfortable 
with soiiie of.the statistical concepts coiiimonly used in the analysis of 
program evaluation data. A secondary purpose is to emphasize the critical 
importance of including data analysis as o^of the resources that mus't be 
used for good program evaluation^ . / 



\ 



Topics Governed In This Section 

• Types of Statistics - Descriptive and Inferential > 

• Measures of Central Tendency 

• Measures of Dispersion or Variability 

• Variance and Testing of Differences between two Measures (Hypothesis Testing) 

• Non-normal Distributions . , 

• Tests for Category Data > 

• Tests for Ordered Data 

' • Tests for Score Data . * 

ThinRB to Stress in Thi's Section - 

1. Descriptive statistics an| inferential statistics serve two different 
purposes. The presentation of program evaluation data to unsophisticated 
audidnces may rely almost entirely on descriptive statistics, but program 
evaluation cannot be considered complete so far as^ the decision maker is 
concerned without inferential tests that tell you whether the change was 
real. 

2. In the coin-flipping demonstration, it is important to convey the idea 
- that any given measure is an estimate! of the true state of affairs. 

Exactness in measurement is not possible in the behavioral realm. .The 
relationship of this demonstration to the concept of randomness is also 
important (see F-3). _^ ■ 

3. Spend sufficient time on the normal distribution (figure at F-12, text at 
F-10 and 13). Explain something about the various scales. 



|b dUlde 



4. 'flexure you draw the distinction between the it'i^nnaJL curve (theoretical 

model:) and how real data distribute th^mselves^b Many data do approximate 
fchis '^npdel, some do not. , : 'J* ^ 

5, The sedition on variance (F-13 through 15) laysSlthe foundation for an 
understanding' of inferential statistics, Take^^enough time so participants ^ 
will get 4n intuitive grasp of why variance is itapo^'tant. 



This section serves as a bri^f inti^duction to statistics and data analysis, 
• We are aware that some of you may hav£ received little formal training in 
this area. We have prepared the exercises, concepts, and examples with 
this in mind. Those of you with experience and training in statistical 
analysis "plqase bear with us.. We hardly expect that those not familiar 
with statist-ics Will learn all about that subject in this session. If you 
leave feeling mo|?e comfortable with the statistical notions expressed - 
'here, feeling^ thl^t you have an intuitive graap of the concepts, that you 
have a better idea of what data analysis can buy you in the way of useful . 
information for evaluatiois and where you can find it, then we will have 
achievied our purpose^. . ^ 



SuRRestions for Teaching^ • . _ 

1. TR. F-0 Tie data anSiysis to other parts of the evaluation process, 

2. TR. F-1 Draw the distinction between descriptive and. inferential 
statistics (F-1). . • 

3. Coin-flipping demonstration (F-2 t^hrough 5) # 

* Rationale: ' \" ^N^,^ 

The coin-f lipping '(^mbnst rat ion 'shows how 

r .- '-^ .' 

• Datarfrom individuals^ can be aggregated for al>group'. 

• Data frotA groups are combine^l for grand totals. 

• Data are org^nizied into frequency distributions. 



Frequency dist^i|)uti^ns are graphically displayed.* 
Various" descriptive ^statistics may be obtained. 
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The first two activities (flipping coins individually and recording 
individual and group results) are carried out by the participants (F-2) • 

After participants have flif)ped their coins and have combined 
their individual tallies with others^at their table, summarize the 
results. * 



Combining Results of Coin-Flipping Demonstration 



Table No. 


ileads 


Talis 


No. Persons 
at Table , 


1 








2 
3 








A 








5 








' ()- 








/ 






^ 8 





KtM'Did t.ihlt' toV.il (MumhiM- ot hivuls, tails, .uul iuiml)iM- oi pi-rsons a{ ^acU 
tahU-), . , 

Sum .1 I L rolmiins I i) y^i^i ^-,1 oup lotals, 

Maki' I lie |)i)int sumuui i i /.i^il al K- i i>n hi)w aitMii atv lijioriMSi^s with lut'i oased 
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Make the points about relationship to randomness suggested in the third 
paragraph at F-3. 

TR> F-3 Move to organizing the data systematically. Record frequencies 
and products. 

The "easiest way to obtain a frequency distribution for the total group is 
to ask how many persons got all 10 heads, how many 9s, 8s, etc, count 
hands, and record these on the transparency. 



Calculate mean, median, and mode, record range (see detail holow and^at 
F-4 of the ^valuator 's Guide). 
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'^^^ ^"^ Lead participa^ies through an exercise in which 'they draiw a line graph 
ot the group's own distribution of h^ads flipped (see F-5 'of the Evaluator's 



Guide). 



Central Tendency, Variability, Non-normal .Dis tr Lbu tions 

SuK^estlons for TeachinR (F-6) 

1. Present mean, median^ and 'mode. 

™' .^"^ Illustration ot bimodal dlstributLon (see F-6) 

^'"^ '^^^^ effect ot extreme scores on the mean (see F-7) 
TR* F--7 -Skewed distributions (see F-8) 

2. Present Range and Standard Deviation.'" 

™' Point out how range uuy vary even though tlu« mean may rtnna Ln" 

TR^U^} Note relationship between size oL standard devLatLou and spread 
TR. F-10 } of scores (see F-IQ and IL), 
. TR. F-11 } / 

3. J R. V^n Discuss tlie normal d b;tr (hii t Lou (siu^ 1-12). Point out' that ()8 
percent ot the s(^ores always lie. between + L standard deviation from, 
the me.ui; 93 percent of the scores Lie between \ 2 standard deviations 
from the mean. Tliis is in.Liio theoretical curve. _l<eal data wlLL approxi- 
mate tlu'se perc(Mitages Lu d L re c t ■ re, La t i on to how much the total dlstributLon 
Ls like the theoretical curve. Review'the percentile equivalents and 

Stan Lues. ihe unequal Intervals iSn both seales can be readily seen. 
I'oiuL out the va r i ous k i uds ol standard scores. ' ' 

^"'-^^A^VA* . M^^ni Standard Deviation 

" ■ ' - ■ • / V 

Ac:c:r loo , :»o 
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Rationale for Using Normal Curve*: If someone questions using the normal 
distribution with aetual distributions whose shapes do not look normal, 
you can explain the theoretical basi\j which allows wide usage of the 
curve, as follows: ' » 



Think of a populaLion with mean M and standard deviation <^ (no matter 
what L t s dis tr ibu t ion) . Draw many independent, random samples of size JN 
♦from the population. Compute the means ^|,xo,X3 . . . ^-q* "^^^^ distribu- 
tion of these means will [>e very nearly normal and ^closely approaches a 
nornvil d i ^it r ibu t ion as more svunpl es are drawn. This distributiou of 
means itself will have the same me^in nnd standard deviation as the . 
ori^^inal population. The coin-flipping demonstration demonstrates this.^ 
T'h Is Is called the Cmitral him it T'heorem and is the inajor reason the 
normal dislrihnlion is u^atl. ' 1 



Prese 


nt 


vai^ iance 


(see l''~13. through IS) . This forms the 
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un(U^ I 
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Infereat;iai Statistics (F-18) 

ThlnRs to Stress ^^--'^^^g.-Sec t ion ' • • 

1- Stress the impP^tance of matching the inferential test to the _kind of 
data at hand. * ^he terms ''score," "ordered," and "category" data are 
^ substitutes for the terms "interval," "ordinal," and "nominal'; data in 
the matheinati<^'^ world. .^^^^y are used'hore because"^^f i negative 
reactions' received in the initial years of the project by individuals 
unfamiliar with conventional ma thenui tical terminology. 

2. Make it clear that thiif ^^^^^^^^.^ of i^^^arential stat Lstics is not intended 
to equip participants to ^^a these, techniqUxi^s if^they have not liad^tevious 
training but to of.fer a sy^aking/ ^t^uaintance with them, For> tho^e 
participants who have ^fS^^^^ ' ^^^^ treatment Is 
designed to pl^;^^- ^^^^ 'varfou.s^^^f^^f • «i&nif icance Ln f^orspective and to 
servo as a levi-^w i)f thefr uses " 

3. ^_Size up your workshop group. Many persons an to mat ic a lily tune out anything 

related to quant it a t ivo. i^^t-)]od^. Try to focus on concepts more than 
caLcufLations. ^t is usu^^j||y i^ot to try to teach compiita tional 

routines. Oi'casloua L fy ^ |. ^^^'^^^^,^^^^^.^,^1 poisons with numerical iacliity will - 
surfacoT The optional Computational sections are designed ior indivadual 
use hy these persons. ^ ^ ^ 

A. The tests i)L -slgnU icauc^. presented are Si)me of ^\^^. more frequently used 
^ ones, paramelL'i^ methods tests of score data, nonpa rame t r ic . me thods. 

^ for tests ol. ordi^red dat;^ aiul categi)ry ilata. 

I 

-^.^ same thinr, ^'^^^^ ^e s.i^^^ about ^;eUHtion ot statist leaf t I'S t s as ahout 

SiMcct Lon ol cvalu<itiou design.s: Ui)lh rcc|ulre attention at a liigh lovi^l 
of li'chnlcal knowUnl^.,.. exposure the I^a r t l,. [ ^^^j,^^ loci'Ives on Iniereutial 

slaLlstlrs hiMi' should liu^i- -j^ j i)r In^r hcMiM' qua lit led to rcco^ni.u. y^^uxi 
skills are uc<'dcd hy iiai^^ analys'ls members dji a pi^of.ram evaluation 
t e .im . 
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Inferential statistics provide a way to test the significance of results 
obtained when data are collected. As noted in the discussion on descrip- 
tive statistics (F~3) , all raeasurjeraent is subject to error "(due to inherent 
difficulties in measuring behavior and to specific testing conditions) and 
to random fluctuation (due to the particular persons included in the' 
sample being tested). Inferential statistics (F-18) provide a way to separate 
chance errors and random fluctuation from ''real" changes. Wliat is^left 
after inferential statistics have been applied are some answers that can be 
given witli confidence on status or change. 



Statistical Tests for Score Data 



Ttiinjt^a to St_ rcss in T his^ Section 

1. Statistical tests for score data are presented only to give exposure to 
tlic participant. Convey the concepts, results, and' applications. 

2. Kncourago par.t-icipaiits to concentrate on tiie uses of these tests of 
statLstLcal sLgnLticance to help them better select program evalu*^tioji 
team members who have the required skills. 



I t tlie data you havo 'can meet the assnmptionj; underlying tests for scoi^' 
data (1-1B)> there avo. many dilLerent and potentLaLLy powerful tests that 
ran be ustnl. Most Ln t e re u t la L t es ts Cor score data require spo.cial training 
for tUeii' proper so Unction and ujie. Un loss the program evalnator has had 
'I li i # I ra La iug, lu* nr y,\xc I .s. advised to stu'k the help oi somt^one who bas . . ^ 

Sevoral commonly nsiul ti'st.s yjii \ bi' mcntionod, Ini t no etiort'\ylll be m^ade 
lo I o.irh rompnt al i ona I rtuitines. Program evalnators who havt* access t(^ a 
I'omnnler ciMitiM" may wish to seek asslstJuco irom that >^onrcc onco the 
tloelsions have boon matle as 'to what kinds oi analyses are .ipp rop ria t e . Do 
not oKpect computer people to hiMp Y^Hi di'c Idiv what: aualysLs i most 
appropriate. 'Ihi'y nuiy i)^. s t at Is ( I an?; af/ we I 1 as ila t a pr oc cssd rr. , bnt 
most .ire not . . - 
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Suggestions for Teach3>fr >^' 

1. Discuss t-tests (F-20), stressing the need to know whigh computational 
routine to use. There are variations depending on the* specif ic data at 



hand. 



Discuss analysis of variance (F-20 and 21), The factorial design^ 
discussed at through 11 represent one use of analysis of variance,' 

Two-way analysis of variance is like a t-test. But analysis of variance 
can be used also when the number W groups beir^g compared . is three or 



mo re, 



^^'^"^'^ Briefly review expanded des igns and factorial poss ibj^i ti es *(C-9 
through 11). Then talk through F-20 and 21 using TR F-17 as the basis 
for discussion. 

3. Discuss multiple regression (F-21). 



Statl'Stical Tests for Ordered Data ' • | 
Things to* S t L'ess _in_ This Secti on 

The four techniques presented here provide a glimpse into tlie world of 
non-parametric statistics. Noii-parametric statistics do not make as 
stringent assumptions about the data as do tests fo'r score data (parametric). 
Solutions are not given except for the Kruskal-Wa Ills Test. Other commonly 
used tests not mentioned are the Mann-Whituey JI Test (a method to compare two 
independent samples) and the Wllcoxon Matched-Pairs Slgned-kank Test (a 
method to compare two correlatld groups). Kxposnre to a few non-p.( rame t r ic 
teehnlcjues Is the objective ot tills d Iscns.s lt)n. 

The ma jov S^ist luct Ions tt) be made when chot)slng among non-parametric 
tests of statLstlcal Ign I f Icantrv are: 

a. How many gronp.s are being compared? In genera I , 'di iioren I tests 
nre used to compare two groups and three or more groups. ^ 
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b. Are the groups independent or are they correlated*? When students 
are'- randomly assigned to different groups, the groups are 
considered independent. If more than one set of ordered data is 
collected on the same students, use statistics for .correlated 
groups. 



Ordered data may be obtained in two ways (F-22) . First: No numerical 
scores are obtained, but you are able to place persons or objects alo^ 
some dimension of interest (as whei^ a committee reviews five textbooks and 
can make decisions in orderitig thera on a scale from most to least preferred) 
Second (and most common): You have obtained test scores but feel thdy 
are not precise enough to meet the assumptions for using statistical 
tests for ^core data. So you order scores from high to low and use that ^ 
ranking ^as ordered data. 



SuRRestions for Teaching ^ 

1. TR> F~18 Demonstrate how to cotrvert score data Uo ranked data (F-22). 
TR. F-19 

2. DLscuss sign test (F-22 and 23). Have participants look at the table on 
page F-23 as you discuss. 

Discuss Rruskai-Wallls Test (F-23 and 2|). Fur those who wish it, the 
solution to this test is given below: 

Solution to tCruska 1-Wal lis Test 

N - 7 H 8 + "9 - 2h ' ' ' 

T - 92 j^'^ 96. ill. 5 - 300 



T ^* 



Solve for Z 
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92^ ^ 96. 5^ . 111.5 



7 ^ 8 



Solve for 



12 



N(N + 1) • 



12 



24(25) 



- .02 



Solve for 3(N I) : 
3(25) = 75 

II = •02(3754.53) - 75 

H - .09 • . 

Refer to X' trabLe of sl^uif Lcaiu'o Lovo Ls (F-26). 
df - (3 - L) - 2 
This result: is not: significant:. 
s 

'4. ■Briofly mention Rank Sums Test" and the Friedman Test (F~24).' 
5. TR. F-2Q re.stzs for Ordered l)at:a (F-24)'.^ 



St:atlsricaL Tests for Category Data 



Suggest: Lons {or ^ ^ L^^'iiliilE. 

1. tsr Introchiee ChL-Square Test (r-25): a test ot deviation f r oia? expec:t ed 

freciuonrles and a test of association 

2. ' Kxplaln Ch.l-Sciuare app I U-a t 1 on for testing deviation fi'oin expected 

frecineneles ^^i^il talk through the example thr(ui};h 27). 



Trainer's Guide 
F-12 



TR, F-21 This first example is based on the following formula 



where f 

o 

.5 is a correction for continuity which is recommended by many current 

statistical treatments of the subject. The correction for con^nuity is 

2. 

needed because a computed X , being based on frequencies (which are whole 
numbers), varies in discrete jumps, whereas the X values given in tables are 
based on a continous scale. 

TR» F-22 Explain again degrees of freedom and levels of significance. Show 
how to read table of significance levels (F-26) , , 

3, Kxplain (.<hi~Square applLcat;ion as'' a test of association (F~27 and 28) . ^ 
TR> F-23 Show how data are displayed for test of iissociatiou. 

4, Present rules which must be followed when using X^ ^>(F-28) • 

Sumnuiry on Data 'Ana Lysis 

Su^^es Lions for l'oacliLn>; 

1. la Ik tlirongh Sumiiuiry on Inferential Statistics 

2. Presi^nL Data InterpreLaL Lun Gnideliues (l'-30 and 31). 

, An jOption 
•1 

TR /F-2A ClompuLat Ion of \' eKeLcLse.s (F-32 through 3')). He .selective. Use 
only It pa rL Lc i[)aii Ls show sii t t Ic*. L ent Inteiest. 11 the group Is divided, 
arrange lor non LnL e res led parllcipanLs to take :i break or hold a coinputa I ion 
» session later lor t ho.sc* who are Interested. 



2 ^ (if -f i -.5)^ 

X = ^ ' Q e' 

f 

e 

= observed frequency 
= expected frequency 
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TR. G-2 

TR. G-3 

TR. G-A 

TR. G-5 

TR. G-6 

TR. G-7 

TR.. G-8 



TRANSPARENCIES 

Sub^jects • 

The Evaluation Process: 
Plan-Conduct-Use - 

Evaluation Results Should 
Be Reported 

Reporting Evaluation Results 

Interim Rep'ort 

Final Report 

Reporting Results 

Reporting Results 

Preparation of the Final 
Report 
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G-1 

G-1 and ^2 
G-2 
G-2 ■ 
G-2 and 3 
G-3 

G-3 



G-1 

G-2 
G-6 



G-6 - 8 



•ARNING KXKRGLSES 



^ ) 
\ 



L() 
17 
18 
19 



RocLpLents and Ujp.es of latorLm EvaUiatLou Data 
netonniiiing Ap prop r La te Data DLspLays 
Vs/ritiug Rerommendat fons for the I'inal Ropyrrt 
Analyzing Ptx^g^'^ii^^ IlvaLuatLou Ri»coiiUinjudat Lous 



G-4 and 5 

- 13 
G~14 - 17 
G-18 - 20 
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In^troduction 

The purposes of this section are to describe the audiences to whom program 
evaluation results are to be reported, the critical importance of tailoring 
reporting, and the several' methods of doing so. This v^rkshop limits itself 
to reporting for local purposes; state and federal programs typically have 
their own evaluation reporting ^requirements. 



TR. G-1. 



Let's review where we are now in the content of the workshop^ 
First, we dealt with planning the evaluation and £ound that 
careful consideration of questions to be answered in final reports 
provides assurance that we are gathering theyright data. Next, we 
discussed conduct ing^tX^"^^A)^^ua^ to collect and analyze data. 
Now, we shall ^atk^ about using tlie evaluation data and getting 
answers to questions raised at the planning, stage. 

\ ■ . 

Suggestions for Tcachinvx ^ ' 

1. ' TR. G-2 Project transparency wlille stating that the . report Lng of evalua- 

tion results is a crucially important aspect of the evaluation process. 
In this seetion, we shall be cons i(lj./r i ng a number nf aud feiiees to whom 
results should be reported. (See C^-l.) 

2. TR. G-3 Cover all subheadings of tU U transparency other than "To WhomV" 
and ask the group to recall them. As the responses; eome from rlie group, 
ask a partli'lpant to record them on a chalkboard or a large pii'ce ot 
paper In front of the room. . ^ 

3. TR. G-3 Now uncover the "Why?^* section on tho t lans pa rency . Kor each 
kind otV t-4^port-reclp1ent named, n^covd on tlie chalkl)oard i he reasons that 
partli:ul/|) ri'clpient: iieiuls t lie report or the uslv; to whieh he or slu' 
might put It.'' ^ X 



/ 



' ^ Note that the ri/'ason.s different audiences neeii to receive program' 

_ ram 



evalnaVion n^ports^s i n ^^imu^ ra I , similar to the [rMrposes ot progr. 

dlseusseil in Stu^tlon A. . i:nu>has I /.e that program evaluation 



evaluation aiseussed in StuM ^ou A. . l.m.a.;/.: J ... 

reports iumuI to be writ:ttMi In terms ol usi's to 4^hii h i*^cipieuts wish to 
put them. 
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Now uncover "When?" section pfi the transparency. State that 



' some audiences need %o make use of program evaluation reports even while 
the program .is .in pro^re^s. The needs of others come into play at the 
end of the program and still others need program evaluation both during 



/ • , - . ■ . : • ■ . f 



and after program activities. 

UAicover the "How?" section of the transparency. State that we now. 
.needito consider p)q4feBi*bic formats for both interim and ead-of -the-year 
, reporting. , . , ^ , 

■ TR. G~4 ' The, eva'lu^Tt^ife\ti report developed while the program is in progress 
is called' the :j[>i.itorlm*'report (see G-2). Continue by paraphrasing the 
infoription on/fhe tf ansp.arency and ' In t\\c. text, 

'^^^L^ '^^^^ final ^report is developed at the end of the program and is, 
among Dlhrr •things, a summary oL the program (r.ee (lontinuo by 

paraphrasing tS<r inL i>rmat ion oii the transparency and in the text, 
TR. G-b Statu that before viewing reporting formats, it might be well to 
review certain consideiat ions applicable to all evaluation reporting. 
Amj) I i 1. y' .some oi I hi* principles I'ittul on the transparency such as: 

• Iute^lIn n^ports must be weLl-timed t\o that there is ample opportunity 
t o'll di'c i s i on ■ maki* r^ Lo modify the proj;r;im or the progrcUii evnluation 
wh'i I e t iii^y ari' in pi'oj\ri*.';s, 

• Report.^ siionUl bi' clearly ami brletly .stati^dl Busy poop le . pi; eL" e r 
iuf oriua t 1 oil that is syntluvi i .:ed anel 'riu-.ordinl ^w.itli chmu' is miens . 

, • I'lu* lonnats ot reports should hi* tMslly undo ij; lood and tavlorod 
( o • i\u' h and i.VMU' e • . • 

Aili'(ina(e riMumn n i r a t i i>u otV proj\iam i*valiiatlon I iil orma t Ion to all 
\yc rs on s pa it i i* I pa t ln>\ i n *i lu' i*va 1 uat i on pr oi- I'ss In a lu^y to 
sui ress. Many piM si>ns aii* l,^:a.rlnl ot [Ui>j',ram I'valuatlon- 11 
I'Vo ryiuu' know:: liOw t lie le sii 1 t s ari* to tu* use d , liowi' ve r , t isi r may 
be il i mi II i s iuul . PiDj'.rtUU i^valuat imi :;liou^d be sihmi as a vehitiW' 
, tor a:;:Vlst i ng I mp I imuo nt im s - l.Ui' I lul I n>', i- I a ss ri>oin tisu'hers — 
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improve prog^ms to b.enefit .children. Direct expec^ience with 
proi^tflaifi evaluations will help diapel.^apprehensi^aJ^that the ^ 
' results ftom such ev^luitl.on3 inight be used 'in s ta|f-evali^aC ion ' 
p»o(jedure8. If'all persohs are involved In on p' way or another in 
the evaluation procesa^and know and agree as to the purpos'es for 
which the results, wiir be used, their fearg will be set aside, 
especially if reports are well done ar^d are widely useful. 
The ptesence of fears as hoted above ^hould not dissuade an 
administration frop its plans to conduct and report prograqi 
evaluations. ■ ^ . - 



TR. ,_G--7 Make a sammary ' statement on' pr inciples of program evaluation 
reporting. ^ V - . " ^ ' ' . . 



8. Discuss the Program Management Review Record on G-3. Point out its 
versatility as both a programs-monitoring* tool and; as ^ in terim, ^valuat fon 
reporting form. * ' , ' 

9. Introduce Learning Exercise 16^ ' Recip ients • a^Uses of Interim Evaluation 
Data (G-4). The purppS'e of . the exercise is to dQtermin-e the kinds of 

^individuals to whom an in tjerim evaluation report>'may be relevant and the 
-uses they might wish to make of- the data r^ported.^ Discuss afCer 8-10 
"minutes. . - . . * , 

10. In end-of-.the-year Pepcy^ng, It is ■ impdrtanrto. keep in mind tlic need bp ' 
report to a variet^^of i(u^iei\ces in a variety of styles and to coasidtjr 
appropriate .formaC^|"for, each type of- audience. 

11. Intr^dMce Learning Exercise l7. Determining Appropriate Datav^^Jtepiays 
(G-9 through 1.3). Discuss after approximately 10 minutes. ' , 

12. TRt G-8 Uncover topic headings one at'a time and discuss the purposeo£ . 
each as describ(?d*^on G-6 - 8 of the Guide . . ^ V 

13. keinfqrce this discussion by asking the pbr t ic Ipants, to read G-'^ . through - 
/8 on a suggested' oii'tUne for the final . rtiport (^^hiAe nUnutes) . - FplloW 

, the reading with a dis^yssion of preparation/ r,ev/ew, comment / rcAsi6n, ' 
^ and approval' procedures: ' / . } ^ ^ 



i 



Who. woul-d likely draft , the-^rogram evalualijLon report? 
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• How might one verify the evaiunt£on findlngfl with the progriim 
participants? . " . 
Wl)o might participate in the review process? 

• What steps wouid lead up to issuing tUif approved report? 

i4. Introduce Learning Exerdise i8, Writing Re commend at ions for the Finai 
Report (G-i4' through i7). As the groups are reading the materials, 
distribute transparencies or large sheets of paper and felt pens to each 
group.' After the participants have ^ad approximately 15 minutes for 
discussion, ask each group to record* its recommendatiorrs to pre^'ent to ^ 
the claus. Call upon spokespersons froifi two or three groups to repoiT't 
their recommendations to the total group. After eactl^ repor t , ask the * 
question: "Aire , those rt/commendations supported by the data?" 



15, State that conclusions :and recommendations in the program evaluation 

report give decision makers b^ses for action. Then ask the participants 
to. read the instructions for Learning Exercise 19, Analyziijg Program 
Evaluation Rccgralniieridat ions (G-18 through 20), ;Each participant^ should 
rate each of the recommendations and -then compare the ' resul ts' o^hJ.'S p. 
analysis with those of another participant. Encourage two-:pers>Cw^t earns 
to discuss the differences they may have. Conclude after approximately 
,^ »«15 minutes. 



16. State that clear, concise reporting is essential to the next step, 
applying '^rog ram evaluation findirlgs. • 
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TR. H-0 The Evaluat ion Process 

TR. H-1 . Sumfiuiry of Steps 

TR. H-2 There is an Alternative 



H-1 * 
H-1 
H-2 ^ 



20 
21 



LEARNING EXERCISES 

Use of Evaluation Information' 
Roadl>locks to Program Evaluation 



H-.2 - 11 
H-12 
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Introduction ,' ' 

The purpose! of this section is to reinforce the point that has been made ^ 
throughout the Guide: that evaluation serves no purpose until it is used. ' 
The primary activity is an exercisfe designed.,' to provide experience in analyzing 
a program evaluation report from tl\e perspective of different audiences and 
to try to make explicit some of its practical uses. 

Suggestons for Teaching 

1. TR. H-0 Integrate the eight elements of program evaluation and focus 
upon applications and uses to-be made of the results. 

2. TR. H-1 Summarize the step^ to tje'taken to accommodate each audiente. '; 

_ The purposes of these steps is ,to give the evaluator direction 'fS? the " 
evaluation plan, to sensitize each audience to its own-needs arid 
responsibilities, and to make all more aware of potential uses of evalua- 
tion information -even before the program begins. 

3. Introduce Learning Exercise 20, Use of Evaluation Information (H-2 
through 11). Hav^articipants work .in groups, arranging with eiich one , 
to rep9rt the findings o|, his or her group during the discussion period.\ 

^ You may haye^^each group 'prepare a transparency or report from notes, 

^ whichever seems most appropriate. The most important part. of the exercise 
is the discussion following the several group reports. Here are some 
questions y'ou may wish to ask: 

• What audiences do you think this report would b| appropriate for? 
Give examples of why you think so. 

What quest^onk do you think the program evaluation might, have " • ' 
addressed? - " ' > 
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Are there parts of the program eVaiuat ion plan that you would 
quarrel with-? Whafc would yj)u ^lave done differently? 

Discuss the nei^c^ for strategic planting to anticipate rgadblocks to 
effective us^^^oL pfogranv-evaluation results and neutiralize them. Introduce 
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^earning Exercise 21, -Roadblocks to Program Evaluation (H-12) . Liat all 
-roadblocks identified. Discuss ways to overcome, circumvent, and.minimi'z 
each obstacle identified. 

5. Conclude the workshop with an optimdstic statement on the 

potential of program evaluation in leading to improvements in a program, 
in the process of ^educational adminis tration, and in education itself. 
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